Add voiceactivity action. #333

jianjunz · 2024-06-28T03:45:10Z

This change adds support for the voice activity detection (VAD) feature for microphones. It allows applications to show a notification when the user is speaking while the MediaStreamTrack is muted.

Related discussion in mediacapture-extensions spec: issue, pr, slide

Preview | Diff

This change adds support for the voice activity detection (VAD) feature for microphones. It allows application to show a notification when user is speaking but MediaStreamTrack is muted.

youennf

Some comments below, looks good overall to me.

We might want to add some text related to toggleMicrophone/Camera/voiceactivity source's target so that these actions are called on MediaSession objects whose context has capture MediaStreamTracks. This should be done in a separate PR though.

youennf · 2024-06-28T06:26:00Z

index.bs

+    <p>
+      A user agent MUST invoke {{MediaSessionActionHandler}} for
+      {{MediaSessionAction/voiceactivity}} only when the voice activity is
+      detected from a source with one or more live {{MediaStreamTrack}}s. A user


We probably want to state that this is restricted to microphone tracks.

Changed "source" to "microphone".

youennf · 2024-06-28T06:27:06Z

index.bs

@@ -1496,6 +1513,7 @@ parameter whose dictionary type is:
  <li>{{MediaSessionActionDetails}} for {{MediaSessionAction/nextslide}}.</li>
  <li>{{MediaSessionActionDetails}} for
  {{MediaSessionAction/enterpictureinpicture}}.</li>
+  <li>{{MediaSessionActionDetails}} for {{MediaSessionAction/voiceactivity}}.</li>
 </ul>

 The <dfn dict-member for="MediaSessionActionDetails">action</dfn>


Maybe we could an example that links voice activity with displaying a UI that can execute setMicrophoneActive(true)

jianjunz · 2024-06-28T09:03:45Z

We might want to add some text related to toggleMicrophone/Camera/voiceactivity source's target so that these actions are called on MediaSession objects whose context has capture MediaStreamTracks. This should be done in a separate PR though.

I also prefer a separate PR for toggleMicrophone/Camera, so this PR doesn't change anything other than voiceactivity action.

chrisn

I left a few comments inline. My bigger question is why expose this via Media Session and not on the media stream track? Also need to think about the privacy implications of allowing pages to detect voice activity on muted tracks.

chrisn · 2024-07-01T10:35:31Z

index.bs

+          the action's intent is to notify the action handler that a voice
+          activity is started.


Could the following be a bit clearer?

Suggested change

the action's intent is to notify the action handler that a voice

activity is started.

the action's intent is to notify the web page that voice

activity has been detected by the microphone.

Added a note to make it clearer.

chrisn · 2024-07-01T10:36:22Z

index.bs

@@ -541,6 +546,17 @@ platform UI or media keys, thereby improving the user experience.
      {{MediaSessionActionHandler}} before running, as different tasks, the
      steps defined to [$set a track's muted state$].
    </p>
+    <p>
+      A user agent MUST invoke {{MediaSessionActionHandler}} for


Suggested change

A user agent MUST invoke {{MediaSessionActionHandler}} for

A user agent MUST invoke the {{MediaSessionActionHandler}} for

chrisn · 2024-07-01T10:36:39Z

index.bs

@@ -541,6 +546,17 @@ platform UI or media keys, thereby improving the user experience.
      {{MediaSessionActionHandler}} before running, as different tasks, the
      steps defined to [$set a track's muted state$].
    </p>
+    <p>
+      A user agent MUST invoke {{MediaSessionActionHandler}} for
+      {{MediaSessionAction/voiceactivity}} only when the voice activity is


Suggested change

{{MediaSessionAction/voiceactivity}} only when the voice activity is

{{MediaSessionAction/voiceactivity}} only when voice activity is

chrisn · 2024-07-01T10:38:59Z

index.bs

+      user agent MAY ignore a {{MediaSessionAction/voiceactivity}} action if all
+      {{MediaStreamTrack}}s associated with the source are not
+      {{MediaStreamTrack/muted}}. It is RECOMMENDED for user agents to set a
+      minimal interval for invoking {{MediaSessionActionHandler}} for


I suggest clarifying "minimal interval" - is this a time delay after voice activity is detected before the action handler is invoked? And how long does voice activity need to be present for? And does voice activity that comes and goes cause multiple invocations? (Not suggesting we spec these things, just clarify what we're recommending user agents to consider)

I suggest clarifying "minimal interval" - is this a time delay after voice activity is detected before the action handler is invoked?

It's intended to be a minimal interval between two voiceactivity actions (or events? event sounds to be more accurate here).

And how long does voice activity need to be present for? And does voice activity that comes and goes cause multiple invocations?

It actually depends on the voice activity detection (VAD) algorithm. Sometimes VAD algorithm may even consider background noise as a voice activity. Based on the use case (unmute microphone notification) we want to target, it is recommended not invoking this action handler too frequently.

A new note section is added for some background about this action.

youennf · 2024-07-01T11:49:11Z

My bigger question is why expose this via Media Session and not on the media stream track?

Good question.
The goal of the action is to allow the web page to ask the user whether they want to unmute microphone capture.
This is done via MediaSession setMicrophoneActive.
Hence why it is best placed at MediaSession level.

There is some desire to expose whether there is voice info in live microphone capture.
This is a different use case that seems best resolved at MediaStreamTrack level, not via MediaSession.

Also need to think about the privacy implications of allowing pages to detect voice activity on muted tracks.

Again, good question.
First, OSes like macOS or iOS added support for muted talker detection (capture is muted but there is still the possibility to know whether user might be talking). This is a good sign that the same functionality is desirable for web applications.

If web applications are not able to help users unmute themselves, web applications will not ask to be muted.
They will instead keep microphone capture running just to detect muted talkers, mute meaning in that case to not send any audio. So we are kept in a world where OS/UA tells microphone is on but Web application tells microphone is off.

FWIW, in Safari, there is a capture indicator telling whether capture is running (red icon), capture is muted (bar gray icon) or capture is not running (no icon). This new action would only happen in the second case, not in the third one (much more problematic).

youennf · 2024-07-01T11:49:27Z

Thanks for putting that to the agenda.

jianjunz

Hi Chris, thanks for your review.

Also thanks to Youenn for providing detailed info about this action, and OS support.

jianjunz · 2024-07-02T06:21:20Z

index.bs

@@ -541,6 +546,17 @@ platform UI or media keys, thereby improving the user experience.
      {{MediaSessionActionHandler}} before running, as different tasks, the
      steps defined to [$set a track's muted state$].
    </p>
+    <p>
+      A user agent MUST invoke {{MediaSessionActionHandler}} for


jianjunz · 2024-07-02T06:22:05Z

index.bs

@@ -541,6 +546,17 @@ platform UI or media keys, thereby improving the user experience.
      {{MediaSessionActionHandler}} before running, as different tasks, the
      steps defined to [$set a track's muted state$].
    </p>
+    <p>
+      A user agent MUST invoke {{MediaSessionActionHandler}} for
+      {{MediaSessionAction/voiceactivity}} only when the voice activity is


jianjunz · 2024-07-02T06:32:33Z

index.bs

+      user agent MAY ignore a {{MediaSessionAction/voiceactivity}} action if all
+      {{MediaStreamTrack}}s associated with the source are not
+      {{MediaStreamTrack/muted}}. It is RECOMMENDED for user agents to set a
+      minimal interval for invoking {{MediaSessionActionHandler}} for


I suggest clarifying "minimal interval" - is this a time delay after voice activity is detected before the action handler is invoked?

It's intended to be a minimal interval between two voiceactivity actions (or events? event sounds to be more accurate here).

And how long does voice activity need to be present for? And does voice activity that comes and goes cause multiple invocations?

It actually depends on the voice activity detection (VAD) algorithm. Sometimes VAD algorithm may even consider background noise as a voice activity. Based on the use case (unmute microphone notification) we want to target, it is recommended not invoking this action handler too frequently.

A new note section is added for some background about this action.

steimelchrome

This seems reasonable to me

chrisn

A few more editorial suggestions, but overall this looks good. Thank you!

chrisn · 2024-07-11T11:09:52Z

index.bs

@@ -412,6 +412,11 @@ platform UI or media keys, thereby improving the user experience.
          the action's intent is to open the media session in a
          picture-in-picture window.
        </li>
+        <li>
+          <dfn enum-value for=MediaSessionAction>voiceactivity</dfn>:
+          the action's intent is to notify the web page that a voice activity


Suggested change

the action's intent is to notify the web page that a voice activity

the action's intent is to notify the web page that voice activity

chrisn · 2024-07-11T11:11:34Z

index.bs

+      agent MAY ignore a {{MediaSessionAction/voiceactivity}} action if all
+      {{MediaStreamTrack}}s associated with the source are not
+      {{MediaStreamTrack/muted}}. It is RECOMMENDED for user agents to set a
+      minimal interval for invoking {{MediaSessionActionHandler}} for


Suggested change

minimal interval for invoking {{MediaSessionActionHandler}} for

minimal interval between invocations of the {{MediaSessionActionHandler}} for

chrisn · 2024-07-11T11:11:52Z

index.bs

+    </p>
+
+    <p class=note>
+      {{MediaSessionAction/voiceactivity}} only indicates the start of a voice


Suggested change

{{MediaSessionAction/voiceactivity}} only indicates the start of a voice

{{MediaSessionAction/voiceactivity}} only indicates the start of voice

chrisn · 2024-07-11T11:12:10Z

index.bs

+
+    <p class=note>
+      {{MediaSessionAction/voiceactivity}} only indicates the start of a voice
+      activity. Application may display a notification if the user is speaking


Suggested change

activity. Application may display a notification if the user is speaking

activity. Applications may display a notification if the user is speaking

chrisn · 2024-07-11T11:13:12Z

index.bs

+      {{MediaSessionAction/voiceactivity}} only indicates the start of a voice
+      activity. Application may display a notification if the user is speaking
+      while the {{MediaStreamTrack}} is muted, or start an {{AudioWorklet}} for
+      audio processing. No action is defined for the end of a voice activity.


Suggested change

audio processing. No action is defined for the end of a voice activity.

audio processing. No action is defined for the end of voice activity.

chrisn · 2024-07-11T11:13:41Z

index.bs

+      activity. Application may display a notification if the user is speaking
+      while the {{MediaStreamTrack}} is muted, or start an {{AudioWorklet}} for
+      audio processing. No action is defined for the end of a voice activity.
+      Unlike other actions which are explicitely triggered by the user,


Suggested change

Unlike other actions which are explicitely triggered by the user,

Unlike other actions which are explicitly triggered by the user,

chrisn · 2024-07-11T11:23:01Z

index.bs

+      efficiency concern, web page may not be notified if the second voice
+      activity started soon after last {{MediaSessionAction/voiceactivity}}
+      action.


Suggested change

efficiency concern, web page may not be notified if the second voice

activity started soon after last {{MediaSessionAction/voiceactivity}}

action.

efficiency concerns, the web page may not be notified if voice

activity ends and restarts soon after the last {{MediaSessionAction/voiceactivity}}

action.

MediaStreamTrack.muted is a readonly attribute. Replace it with enabled, which can be set by the application.

jianjunz

Thanks for your comments.

jianjunz · 2024-07-12T07:44:37Z

index.bs

@@ -412,6 +412,11 @@ platform UI or media keys, thereby improving the user experience.
          the action's intent is to open the media session in a
          picture-in-picture window.
        </li>
+        <li>
+          <dfn enum-value for=MediaSessionAction>voiceactivity</dfn>:
+          the action's intent is to notify the web page that a voice activity


jianjunz · 2024-07-12T07:44:46Z

index.bs

+      agent MAY ignore a {{MediaSessionAction/voiceactivity}} action if all
+      {{MediaStreamTrack}}s associated with the source are not
+      {{MediaStreamTrack/muted}}. It is RECOMMENDED for user agents to set a
+      minimal interval for invoking {{MediaSessionActionHandler}} for


jianjunz · 2024-07-12T07:44:53Z

index.bs

+    </p>
+
+    <p class=note>
+      {{MediaSessionAction/voiceactivity}} only indicates the start of a voice


jianjunz · 2024-07-12T07:50:11Z

index.bs

+
+    <p class=note>
+      {{MediaSessionAction/voiceactivity}} only indicates the start of a voice
+      activity. Application may display a notification if the user is speaking


jianjunz · 2024-07-12T07:50:19Z

index.bs

+      {{MediaSessionAction/voiceactivity}} only indicates the start of a voice
+      activity. Application may display a notification if the user is speaking
+      while the {{MediaStreamTrack}} is muted, or start an {{AudioWorklet}} for
+      audio processing. No action is defined for the end of a voice activity.


jianjunz · 2024-07-12T07:50:27Z

index.bs

+      activity. Application may display a notification if the user is speaking
+      while the {{MediaStreamTrack}} is muted, or start an {{AudioWorklet}} for
+      audio processing. No action is defined for the end of a voice activity.
+      Unlike other actions which are explicitely triggered by the user,


jianjunz · 2024-07-12T07:50:34Z

index.bs

+      efficiency concern, web page may not be notified if the second voice
+      activity started soon after last {{MediaSessionAction/voiceactivity}}
+      action.


jianjunz · 2024-07-12T07:54:59Z

index.bs

+      from a microphone with one or more live {{MediaStreamTrack}}s. A user
+      agent MAY ignore a {{MediaSessionAction/voiceactivity}} action if
+      microphone is not muted and all {{MediaStreamTrack}}s associated with the
+      source are {{MediaStreamTrack/enabled}}. It is RECOMMENDED for user agents


Replaced MediaStreamTrack.muted with MediaStreamTrack.enabled because muted is a readonly property. Application may want to set enabled to true if voice activity is detected.

youennf

This looks good to me.
I think we can add some wording stating the voiceactivity action source MUST always have a target whose document MUST always have live microphone {{MediaStreamTrack}}s. This will more closely reuse spec wording and tighten the fact that voiceacitvity cannot happen on a document that is not capturing microphone

chrisn

Sorry, a few more minor comments.

chrisn · 2024-07-12T08:20:23Z

index.bs

+      {{MediaSessionAction/voiceactivity}} only when voice activity is detected
+      from a microphone with one or more live {{MediaStreamTrack}}s. A user
+      agent MAY ignore a {{MediaSessionAction/voiceactivity}} action if
+      microphone is not muted and all {{MediaStreamTrack}}s associated with the


Suggested change

microphone is not muted and all {{MediaStreamTrack}}s associated with the

the microphone is not muted and all {{MediaStreamTrack}}s associated with the

chrisn · 2024-07-12T08:28:09Z

index.bs

+    <p>
+      A user agent MUST invoke the {{MediaSessionActionHandler}} for
+      {{MediaSessionAction/voiceactivity}} only when voice activity is detected
+      from a microphone with one or more live {{MediaStreamTrack}}s. A user


If "live" here is the same as {{MediaStreamTrackState/live}}, we could link them:

Suggested change

from a microphone with one or more live {{MediaStreamTrack}}s. A user

from a microphone with one or more {{MediaStreamTrackState/live}} {{MediaStreamTrack}}s. A user

chrisn · 2024-07-12T08:49:16Z

index.bs

+      A user agent MUST invoke the {{MediaSessionActionHandler}} for
+      {{MediaSessionAction/voiceactivity}} only when voice activity is detected
+      from a microphone with one or more live {{MediaStreamTrack}}s. A user
+      agent MAY ignore a {{MediaSessionAction/voiceactivity}} action if


"Ignore" seems strange here and implies the action is generated somehow, but ignored. Could we say "A user agent MAY ignore voice activity if the microphone is not muted ..."?

Updated. It sounds better.

jianjunz · 2024-07-15T02:40:30Z

This looks good to me. I think we can add some wording stating the voiceactivity action source MUST always have a target whose document MUST always have live microphone {{MediaStreamTrack}}s. This will more closely reuse spec wording and tighten the fact that voiceacitvity cannot happen on a document that is not capturing microphone

Thanks for the suggestion. It's added to the beginning of the paragraph.

chrisn

Looks good, thank you!

jianjunz

Looks good, thank you!

Thanks for your review. Please help merge it if it's okay to merge.

jianjunz · 2024-07-15T02:36:04Z

index.bs

+      A user agent MUST invoke the {{MediaSessionActionHandler}} for
+      {{MediaSessionAction/voiceactivity}} only when voice activity is detected
+      from a microphone with one or more live {{MediaStreamTrack}}s. A user
+      agent MAY ignore a {{MediaSessionAction/voiceactivity}} action if


Updated. It sounds better.

jianjunz · 2024-07-15T02:36:12Z

index.bs

+      {{MediaSessionAction/voiceactivity}} only when voice activity is detected
+      from a microphone with one or more live {{MediaStreamTrack}}s. A user
+      agent MAY ignore a {{MediaSessionAction/voiceactivity}} action if
+      microphone is not muted and all {{MediaStreamTrack}}s associated with the


jianjunz · 2024-07-15T02:36:19Z

index.bs

+    <p>
+      A user agent MUST invoke the {{MediaSessionActionHandler}} for
+      {{MediaSessionAction/voiceactivity}} only when voice activity is detected
+      from a microphone with one or more live {{MediaStreamTrack}}s. A user


chrisn · 2024-07-15T15:54:09Z

Thanks! As they're the editors, I leave @youennf or @steimelchrome to merge.

youennf · 2024-07-18T14:25:50Z

Merging given comments have been addressed.

SHA: 0f6e693 Reason: push, by youennf Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

Add voiceactivity action.

298c998

This change adds support for the voice activity detection (VAD) feature for microphones. It allows application to show a notification when user is speaking but MediaStreamTrack is muted.

jianjunz mentioned this pull request Jun 28, 2024

Add MediaStreamTrack voice activity detection support. w3c/mediacapture-extensions#153

Closed

youennf requested review from steimelchrome and youennf and removed request for steimelchrome June 28, 2024 06:23

youennf reviewed Jun 28, 2024

View reviewed changes

dontcallmedom linked an issue Jun 28, 2024 that may be closed by this pull request

Consider adding onVoiceActivity event on MediaStreamTrack for audio w3c/mediacapture-extensions#145

Closed

jianjunz added 2 commits June 28, 2024 16:51

Add an example and restrict audio source to be micrphone.

35afdfa

Call setMicrophoneActive(true) if user allows to unmute.

2bb2843

chrisn added the agenda Topic should be discussed in a group call label Jul 1, 2024

chrisn reviewed Jul 1, 2024

View reviewed changes

Add a note for voice activity explanation.

7240fbe

jianjunz commented Jul 2, 2024

View reviewed changes

steimelchrome approved these changes Jul 9, 2024

View reviewed changes

chrisn reviewed Jul 11, 2024

View reviewed changes

jianjunz added 2 commits July 12, 2024 10:54

Address comments.

6b22a34

Update the condition when UA may ignore voiceactivity.

f7ebd3d

MediaStreamTrack.muted is a readonly attribute. Replace it with enabled, which can be set by the application.

jianjunz commented Jul 12, 2024

View reviewed changes

youennf approved these changes Jul 12, 2024

View reviewed changes

chrisn reviewed Jul 12, 2024

View reviewed changes

Add some wording for live microphone tracks and some minor fixes.

be55bde

chrisn approved these changes Jul 15, 2024

View reviewed changes

jianjunz commented Jul 15, 2024

View reviewed changes

youennf merged commit 0f6e693 into w3c:main Jul 18, 2024
2 checks passed

github-actions bot added a commit that referenced this pull request Jul 18, 2024

Add voiceactivity action. (#333)

27dba30

SHA: 0f6e693 Reason: push, by youennf Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

chrisn removed the agenda Topic should be discussed in a group call label Dec 5, 2024

		the action's intent is to notify the action handler that a voice
		activity is started.

	A user agent MUST invoke {{MediaSessionActionHandler}} for
	A user agent MUST invoke the {{MediaSessionActionHandler}} for

	{{MediaSessionAction/voiceactivity}} only when the voice activity is
	{{MediaSessionAction/voiceactivity}} only when voice activity is

	the action's intent is to notify the web page that a voice activity
	the action's intent is to notify the web page that voice activity

	minimal interval for invoking {{MediaSessionActionHandler}} for
	minimal interval between invocations of the {{MediaSessionActionHandler}} for

	activity. Application may display a notification if the user is speaking
	activity. Applications may display a notification if the user is speaking

	audio processing. No action is defined for the end of a voice activity.
	audio processing. No action is defined for the end of voice activity.

	Unlike other actions which are explicitely triggered by the user,
	Unlike other actions which are explicitly triggered by the user,

	microphone is not muted and all {{MediaStreamTrack}}s associated with the
	the microphone is not muted and all {{MediaStreamTrack}}s associated with the

	from a microphone with one or more live {{MediaStreamTrack}}s. A user
	from a microphone with one or more {{MediaStreamTrackState/live}} {{MediaStreamTrack}}s. A user

Add voiceactivity action. #333

Add voiceactivity action. #333

Conversation

jianjunz commented Jun 28, 2024 • edited by pr-preview bot Loading

youennf left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jianjunz commented Jun 28, 2024

chrisn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chrisn Jul 1, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

youennf commented Jul 1, 2024

youennf commented Jul 1, 2024

jianjunz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

steimelchrome left a comment

Choose a reason for hiding this comment

chrisn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jianjunz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

youennf left a comment

Choose a reason for hiding this comment

chrisn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jianjunz commented Jul 15, 2024

chrisn left a comment

Choose a reason for hiding this comment

jianjunz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chrisn commented Jul 15, 2024

youennf commented Jul 18, 2024

jianjunz commented Jun 28, 2024 •

edited by pr-preview bot

Loading

chrisn Jul 1, 2024 •

edited

Loading